"Software reliability" refers to the likelihood of software performing in the desired fashion without unexpectedly terminating. Measuring software reliability is a useful benchmark for deciding when a software product is ready for shipment to customers. In addition, software reliability is a useful metric for determining when to use a software product. In certain critical environments, software should not be used unless software reliability is quite high.
Many software product reliability metrics have been used in the computer industry. One such software reliability metric is the up-time of the software product. The up-time refers to the length of time that the software product has executed before the software product failed.
Software reliability metrics are typically obtained by testing the software product to yield performance data which are used to calculate the software reliability metrics. In addition to yielding performance data, testing identifies problems in the software product, which can then be fixed. One method of testing a software product is to use an automated testing system. An automated testing system is preferable to using human testers because an automated testing system can test continuously for extended periods of time and exactly as instructed. On the other hand, a human tester typically can only test continuously for shorter periods of time, and a human tester typically makes inadvertent mistakes which disrupt the testing process.
FIG. 1 depicts a block diagram of a conventional automated testing system 100. The automated testing system 100 holds a copy of the software product 102, a copy of a test program 104, a script 108, and a log file 106. The software product 102 is the software that is being tested by the automated testing system 100. The test program 104 is a program that is run to test the software product 102. The testing is conducted by sending various commands to the software product 102 to invoke the functions of the software product. The script 108 is a batch file that creates commands for the test program 104 to execute on the software product 102. The script 108 contains a random number generator and an identifier of the commands to be tested in the software product 102. The script 108 invokes the random number generator using a seed. A seed is a number which is input into a random number generator for the creation of random numbers. The output of a random number generator is a deterministic sequence of numbers. Therefore, if a seed is used twice by the same random number generator, the same sequence of numbers are generated by the random number generator. The log file 106 is used by the script 108 to store the seed for the random number generator.
In order to test the software product 102 using the conventional automated testing system 100, the test program 104 is invoked by the script 108. The script 108 generates a random number and uses the generated random number to access a list of commands for the software product 102. The list of commands used by the script 108 is a predetermined list of commands that the software product 102 executes. The random number acts as an index into the list of commands whereby the index identifies which command is to be executed in the list. After using the random number to identify a command to execute, the script 108 invokes the test program 104 to execute the command in the software product 102. The test program 104 translates the command from the script 108 into a series of keystrokes or other input that the software product 102 understands. The test program 104 then performs the input command.
The script 108 continues to generate random numbers and execute commands in the above-described fashion until the software product 102 fails. If the software product 102 does not fail, after a predetermined amount of time using one seed, another seed is chosen. The newly chosen seed is then used to generate random numbers that are then used to indicate commands for testing as described above. Each time that a new seed is chosen by the script 108, the script 108 enters the seed into the log file 106. Therefore, when the software product 102 fails, the user can view the log file 106 to determine which seed was used when the software product failed (this is known as "the failure seed"). Since the random number generator generates numbers in a deterministic fashion, the user can enter the failure seed into the script 108 and have the steps that led up to the failure replicated. However, if, for example, the failure occurs three hours after using the failure seed, the failure takes three hours to replicate because the entire sequence of random numbers generated by the random number generator prior to the failure must be again generated to recreate the failure sequence. Replicating the steps that led to a failure is desirable so that the problems in the software that caused the failure can be isolated and fixed. A tester cannot simply jump to the failure point without first reenacting the commands that preceded the failure.
Software products are also typically tested by professional human testers. The professional testers perform many of the same types of tests that are performed by the automated testing system 100. One difficulty with data obtained by professional testers is that it does not accurately reflect how users utilize the software product. Generally, professional testers tend to test parts of the program that many users do not frequently use. As a result, it is difficult to know how to apply the resulting tester data in order to obtain a reasonable measure of software reliability.